Mutual Information and Bayes Methods for Learning a Distribution

Each parameter w in an abstract parameter space W is associated with a di er ent probability distribution on a set Y A parameter w is chosen at random from W according to some a priori distribution on W and n conditionally indepen dent random variables Y n Y Yn are observed with common distribution determined by w Viewing W as a random variable we obtain bounds on the mutual information between the random variable W giving the choice of pa rameter and the random variable Y n giving the sequence of observations This quantity is the cumulative risk in predicting Y Yn under the log loss minus the risk if the true parameter w is known The upper bounds are stated in terms of the Laplace transform of the rate of growth of the volume of relative entropy neighborhoods in the parameter space W and the lower bounds are given in terms of the corresponding quantity using Hellinger neighborhoods We show how these bounds can be interpreted in terms of an average local dimension of the parameter space W under suitable conditions